Skip to content

feat: emit and seed the evo snapshot section in dumptxoutset and snapshot load - #7601

Draft
PastaPastaPasta wants to merge 15 commits into
dashpay:developfrom
PastaPastaPasta:feat/evo-snapshot-load
Draft

feat: emit and seed the evo snapshot section in dumptxoutset and snapshot load#7601
PastaPastaPasta wants to merge 15 commits into
dashpay:developfrom
PastaPastaPasta:feat/evo-snapshot-load

Conversation

@PastaPastaPasta

@PastaPastaPasta PastaPastaPasta commented Aug 13, 2026

Copy link
Copy Markdown
Member

Issue being fixed or feature implemented

Stacked on #7592 and #7593 — every commit below the final one belongs to those PRs (rebased onto develop post-#7603); this PR adds only the final commit. Final PR of the AssumeUTXO M4 decomposition (#7579 — series map there): the lifecycle wiring that makes the format real. dumptxoutset emits the canonical evo section alongside the UTXO set; snapshot load decodes, validates, and seeds it; completion cross-checks everything the earlier PRs promised (retained-section CbTx verification, historical MN-list reconstruction, seeded-modifier integrity) and the assumeutxo prune lock keeps the base block readable until that happens.

What was done?

  • dumptxoutset appends the marker/version/payload/hash evo section; load reads it (AutoFile::size() bounds the decode), validates context-free and against chain, and seeds EvoDB through the feat: build, chain-validate, and seed evo snapshot state #7593 seeding surface without publishing into shared caches. The section is retained in EvoDB (EVODB_SNAPSHOT_EVO_SECTION) for the deferred completion-time CbTx cross-check.
  • Background validation records MN-list hashes only at the snapshot base and the bounded set of required historical work blocks; completion compares them against payload-derived reconstruction — this is what turns the M3-era marker scaffolding into a real integrity check.
  • Runtime seeded-state mismatches (GetAllQuorumMembers) route into the controlled invalid-snapshot shutdown (HandleSnapshotStateMismatch), completing the plumbing feat: build, chain-validate, and seed evo snapshot state #7593 deliberately left unwired.
  • The assumeutxo prune lock (ProtectSnapshotBaseFromPruning/release, BlockManager::DeletePruneLock) — the remainder of the original B6 commit, placed with its consumer as discussed in backport: assumeutxo M4 — evo snapshot format v3 and LLMQ reconstruction #7579 review.
  • The chain-facing helpers this PR adds (seeding, base-block CbTx verification) live in the header-less evo/snapshot_chain.cpp unit from feat: build, chain-validate, and seed evo snapshot state #7593, keeping the codec cycle-free. Keeping ChainstateManager code in validation.cpp makes validation depend on evo/snapshot.h, whose value-type members pull in creditpool/mnhftx/commitment — those three cycles are added to EXPECTED_CIRCULAR_DEPENDENCIES, the same accepted class as the existing validation hub entries.
  • Per that review, every ChainstateManager body stays in validation.cpp — the previously proposed evo/snapshot_load.cpp file is gone from the series; chain-facing helpers are declared in evo/snapshot.h and implemented where upstream keeps the logic.
  • Tests: feature_assumeutxo_dash.py (dump → load → background-validate → complete, plus recovery/invalid paths), rpc_dumptxoutset.py update, and the unit-test deltas for the lifecycle (prune-lock survival, soft-fail base detection, EvoDB retention).

How Has This Been Tested?

Full unit suite green on a --enable-werror build; feature_assumeutxo_dash.py, rpc_dumptxoutset.py, and feature_reindex.py pass locally. The stack beneath it carries its own sanitizer verification.

Breaking Changes

None. dumptxoutset output gains the evo section (new format version); old snapshots without it are rejected at load on DIP3-active chains, which is the intended security posture — there is no legacy Dash snapshot format in the wild.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

PastaPastaPasta and others added 8 commits August 19, 2026 23:15
…e validation

First code PR of the assumeutxo M4 series (dashpay#7579 decomposition): the versioned interchange format for Dash's evo state alongside a UTXO snapshot - canonical serialization, DoS-bounded validating decode, and every validation invariant that needs no chain context. Chain-aware building/validation and dump/load integration follow in the next PRs of the series.

Canonical ordering exists because snapshot content is hashed and cross-checked; per-object serializers are reused through a bounded stream wrapper, with bespoke code only at container level (ordering, bounds, per-entry budgets); decode-time checks deliberately stay out of the trusted hot EvoDB deserializers. AssumeutxoData gains the EvoSnapshotHash anchor the format is pinned by. Includes the aggregate rotation skip-list bound (lists accumulate across every quorum index and wrap the combined MN list), the CRangesSet bounded unserializer, and a vendored-immer shift-base ubsan suppression reachable only through the deliberately hash-colliding test fixtures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… sign change

The mask for rejecting out-of-range trailing bits promotes through operator~ to a negative int before its implicit conversion back to uint8_t, which clang's implicit-integer-sign-change check reports for every bitset whose size is not a multiple of eight. The evo snapshot unit tests are the first to deserialize such bitsets under the sanitizer job. Same bits, stated explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The codec PR shipped UnserializeBounded without its unit coverage; add the malformed/canonical decode matrix and the round-trip checks from the original series.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The signal map normalizes iteration order, so wire order was unobservable after decode: any permutation of the same signal set was accepted by Unserialize() and reserialized sorted, violating the require_canonical_order contract that every other top-level collection enforces. Require the strictly ascending bit order the serializer emits at the only point where wire order is visible, which also subsumes the duplicate-bit rejection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…LMQ size

-llmqtestparams and -llmqdevnetparams mutate size, minSize, threshold, and dkgBadVotesThreshold on the CChainParams copy of the LLMQ table, so a commitment produced under a supported non-default quorum size was rejected by the static Consensus::available_llmqs sizing: a larger size exceeded the read bound and a smaller one failed the exact comparison. The context-free layer now enforces an allocation ceiling (EVO_SNAPSHOT_MAX_QUORUM_SIZE) plus signers/validMembers internal consistency; exact sizing already happens in the chain-aware validation, which iterates the effective consensus.llmqs table and calls VerifySizes against it. Count, rotation, and interval fields stay on the static table since the overrides cannot change them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Validate() bounded only the MNHF map's cardinality: out-of-money-range credit pool amounts, a currentLimit above locked, signal bits at or above VERSIONBITS_NUM_BITS, and signal heights outside [0, base height] all received a canonical snapshot hash. ConstructCreditPool guarantees 0 <= currentLimit <= locked in every deployment branch and consensus admits MNHF signals only for bits below VERSIONBITS_NUM_BITS at their mined height, so enforce exactly those invariants.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PastaPastaPasta and others added 2 commits August 20, 2026 01:21
Unserialize() appended to quorums, historical_mn_list_diffs, and quorum_modifiers and merged into the existing MNHF signal map, so a successful decode into a reused object accumulated state the consumed bytes never contained and could still pass Validate(). Clear the collections up front.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CDeterministicMNList's HAMT hashes proTxHash by its first 8 bytes, so a snapshot supplying up to 100,000 distinct, canonically ordered hashes sharing one 64-bit prefix made every AddMN copy the whole immer collision node: quadratic work and allocation from a single crafted snapshot. Real proTxHashes are uniform txids, where even one shared prefix among 100,000 has probability ~3e-10, so bound collision runs at 8. Enforced on the sorted base list during decoding (before the inserts), on the merged current-plus-additions prefix set before every historical diff application, and as an object-level invariant; the run detector sorts a plain vector so it cannot itself be driven into hash-collision buckets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PastaPastaPasta and others added 5 commits August 20, 2026 08:33
The context-free CbTx cross-check compared the MN root, active-quorum root, and credit-pool balance but never the height, so a CbTx claiming a different height passed whenever those values were unchanged; the test even verified successfully with nHeight left at zero against a height-500 list. Compare cbtx.nHeight with the snapshot list's height.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The top-level decoder always constructs a fresh local, but direct stream >> data into a reused CQuorumSnapshotData retained prior commitments and rotation entries and could exceed the incoming count bounds. Clear the three vectors up front, matching CEvoSnapshot::Unserialize.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-diff operation budget charges only additions, updates, and removals, so a few hundred bytes of zero-operation diff entries could drag a maximum-size list across the whole table-wide history horizon: every entry traverses, sorts, and canonically hashes the full reconstructed list (~19M record serializations from a small input). Charge a cumulative record budget up front in ReconstructHistoricalMNLists. The table-wide horizon sums types no single network enables together, so even a ceiling-sized list under a fully loaded real configuration stays well below half the budget.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second code PR of the assumeutxo M4 series: BuildEvoSnapshot() collects commitments, rotation snapshots, historical MN-list diffs, and exact score modifiers from chain state; ValidateEvoSnapshotAgainstChain() verifies a decoded snapshot against the block index, deployments, and quorum reconstruction; and the EvoDB seeding surface writes exactly the state reconstruction later reads. The chain-aware implementations live in the header-less evo/snapshot_chain.cpp unit so the codec in evo/snapshot.cpp never includes validation.h (no evo/snapshot -> validation cycle); ChainstateManager code is untouched here and stays in validation.cpp later in the series.

Parameter-derived quorum counts are carried as maxima rather than exact requirements: a young chain, a freshly activated type, or a rotation type whose horizon predates activation legitimately has less history, and requiring the full horizon made dumptxoutset fail on valid chain state (flagged as blocking by review on the original M4 PR). Completeness is anchored by the completion-time CbTx quorum merkle root, the historical-diff and modifier tallies, and completion-time quorum reconstruction; two regression tests fail against exact-count enforcement. GetHashModifier() prefers a seeded exact modifier when work-block data is unavailable and cross-checks it when present; the mismatch error stays unreachable until the load integration seeds modifiers. GetDeterministicMNListHash() switches to the canonical codec hash (dev-channel-only marker-compat caveat in the PR description).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…shot load

Final PR of the assumeutxo M4 series: dumptxoutset appends the canonical evo section (marker, version, payload, hash) after the UTXO records; snapshot activation decodes it, validates it context-free and against the chain, seeds EvoDB through the seeding surface, retains the section for the deferred completion-time CbTx cross-check, and records background MN-list hashes only for the snapshot base and the bounded set of historical work blocks. The completion path cross-checks reconstructed historical MN lists, verifies the retained section against the base CbTx, and routes runtime seeded-state mismatches (GetAllQuorumMembers) into the controlled invalid-snapshot shutdown. The assumeutxo prune lock keeps the base block's data available until that check completes.

Per review on the original M4 PR, every ChainstateManager body stays in validation.cpp: the previously proposed evo/snapshot_load.cpp move is gone, with the chain-facing helpers declared in evo/snapshot.h and the load/build/completion logic living where upstream keeps it. feature_assumeutxo_dash.py exercises the dump/load/complete cycle end to end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant